Boxplots

Boxplots#

This page contains instructions and documentation for creating plots used to visualize curve ensembles.

spaghetti_plot#

Plots a random selection of curves.

Parameters#

client (bigquery.Client): BigQuery client object.

table_name (str): BigQuery table name containing data in ‘dataset.table’ form.

reference_table (str): BigQuery table name containing reference table in ‘dataset.table’ form.

geo_level (str): The name of a column from the reference table. The geographical level used to determine what places are included.

geo_values (str or listlike or None): The source(s) to be included. A value or subset of values from the geo_level column. If None, then all values will be included.

geo_column (str, optional): Name of column in original table containing geography identifier. Defaults to ‘basin_id’.

reference_column (str, optional): Name of column in original table containing the geography corresponding to data in source_column and target_column. Defaults to ‘basin_id’.

value (str, optional): Name of column in the original table containing the importation value to be analyzed. Defaults to ‘value’.

n (int, optional): Number of curves to plot. Defaults to 25.

Returns#

fig (plotly.graph_objects.Figure): Plotly Figure containing visualization.

Example#

import epidemic_intelligence as ei
from google.oauth2 import service_account
from google.cloud import bigquery

credentials = service_account.Credentials.from_service_account_file('../../../credentials.json') # use the path to your credentials
project = 'net-data-viz-handbook' # use your project name
# Initialize a GC client
client = bigquery.Client(credentials=credentials, project=project)

table_name = 'h1n1_R2.basins_prevalence_agg'
reference_table = 'reference.gleam-geo-map'
reference_column = 'basin_id' # name of a column in reference table
geo_column = 'basin_id' # name of a column in table corresponding to column in reference table
geo_level = 'basin_label' 
geo_values = 'Portland(US-ME)' 
value = 'Infectious_18_23'

sp_fig = ei.spaghetti_plot(
    client=client,
    table_name=table_name,
    reference_table=reference_table,
    geo_level=geo_level,
    geo_values=geo_values,
    geo_column=geo_column,
    reference_column=reference_column,
    value=value,
    n=100)

# finishing touches
sp_fig.update_layout(width=900, height=500, 
                     showlegend=True, 
                     font_family='PT Sans Narrow', 
                     title='Spaghetti Plot',)
sp_fig.show()

functional_boxplot#

A functional boxplot uses curve-based statistics that treat entire curves as a single data point, as opposed to each observation in a curve. Always plots the median and interquartile range.

Parameters#

client (bigquery.Client): BigQuery client object.

table_name (str): BigQuery table name containing data in ‘dataset.table’ form.

reference_table (str): BigQuery table name containing reference table in ‘dataset.table’ form.

geo_level (str): The name of a column from the reference table. The geographical level used to determine what places are included.

geo_values (str or listlike or None): The source(s) to be included. A value or subset of values from the geo_level column. If None, then all values will be included.

geo_column (str, optional): Name of column in original table containing geography identifier. Defaults to ‘basin_id’.

reference_column (str, optional): Name of column in original table containing the geography corresponding to data in source_column and target_column. Defaults to ‘basin_id’.

value (str, optional): Name of column in the original table containing the importation value to be analyzed. Defaults to ‘value’.

num_clusters (int, optional): Number of clusters that curves will be broken into based on grouping_method. Defaults to 1. Note: raising num_clusters above one significantly increases runtime.

num_features (int, optional): Number of features the kmeans algorithm will use to group curves if num_clusters in greater than 1. Must be less than or equal to number of run_ids in table.

grouping_method (str, optional): Method used to group curves. Must be one of:

'mse' (default): Fixed-time pairwise mean squared error between curves.
'abc': Fixed-time pairwise area between curves. Also called mean absolute error.

kmeans_table (str, optional): BigQuery table name containing clustering information in ‘dataset.table’ form. Used when kmeans has already been performed with delete_data=False. Allows function to skip costly kmeans algorithm.

centrality_method (str, optional): Method used to determine curve centrality within their group. Must be one of:

'mse' (default): Summed fixed-time mean squared error between curves.
'abc': Summed fixed-time pairwise area between curves. Also called mean absolute error.
'mbd': Modified band depth. For more information, see Sun and Genton (2011).

threshold (float, optional): Number of interquantile ranges from median curve must be to not be considered an outlier. Defaults to 1.5.

dataset (str or None, optional): Name of BigQuery dataset to store intermediate tables. If None, then random hash value will be used. Defaults to None.

delete_data (bool, optional): If True, then intermediate data tables will not be deleted. Defaults to False.

overwrite (bool, optional): If True, then will not prompt for confirmation if overwriting an existing BigQuery dataset. Defaults to False.

Returns#

fig (plotly.graph_objects.Figure): Plotly Figure containing visualization.

Example#

# required
table_name = 'h1n1_R2.basins_prevalence_agg'
reference_table = 'reference.gleam-geo-map'
reference_column = 'basin_id' # name of a column in reference table
geo_column = 'basin_id' # name of a column in table corresponding to column in reference table
geo_level = 'basin_label'
geo_values = 'Portland(US-ME)'
value = 'Infectious_18_23'

# Set parameters for grouping
num_clusters = 1
num_features = 20 
grouping_method = 'mse' # mean squared error
centrality_method = 'mse' # mean squared error

dataset = None
delete_data = True

fbp_fig = ei.functional_boxplot(
    client=client,
    table_name=table_name,
    reference_table=reference_table,
    geo_level=geo_level,
    geo_values=geo_values,
    geo_column=geo_column,
    reference_column=reference_column,
    value=value,
    num_clusters=num_clusters,
    num_features=num_features,
    grouping_method=grouping_method,
    centrality_method=centrality_method,
    dataset=dataset,
    delete_data=delete_data,
    overwrite=True
)

# finishing touches
fbp_fig.update_layout(width=900, height=500, 
                     showlegend=True, 
                     font_family='PT Sans Narrow', 
                     title='Functional Boxplot',
                     yaxis_title="Infectious 18-23yo"
)
fbp_fig.show()

Dataset `net-data-viz-handbook.717fb4a83d0d2c670816798c0103232eeb31674b262db45377e495c89164ffd1` created.

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[2], line 19
dataset = None
delete_data = True
---> 19 fbp_fig = ei.functional_boxplot(
   client=client,
   table_name=table_name,
   reference_table=reference_table,
   geo_level=geo_level,
   geo_values=geo_values,
   geo_column=geo_column,
   reference_column=reference_column,
   value=value,
   num_clusters=num_clusters,
   num_features=num_features,
   grouping_method=grouping_method,
   centrality_method=centrality_method,
   dataset=dataset,
   delete_data=delete_data,
   overwrite=True
)
# finishing touches
fbp_fig.update_layout(width=900, height=500, 
                    showlegend=True, 
                    font_family='PT Sans Narrow', 
                    title='Functional Boxplot',
                    yaxis_title="Infectious 18-23yo"
)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\epidemic_intelligence\boxplots.py:338, in functional_boxplot(client, table_name, reference_table, geo_level, geo_values, geo_column, reference_column, value, num_clusters, num_features, grouping_method, kmeans_table, centrality_method, threshold, dataset, delete_data, overwrite)
# Step 8
get_mid_curves = f"""-- Step 3: Calculate min and max values at each time step using the non-outliers table
   SELECT
       data.date,
   (...)
       date;
   """
--> 338 plt_middle = client.query(get_mid_curves).to_dataframe()  # Execute and fetch results
get_outliers = f"""-- Step 3: Calculate min and max values at each time step using the non-outliers table
   WITH outliers AS (
       SELECT 
   (...)
       date;
   """
plt_outliers = client.query(get_outliers).to_dataframe()  # Execute and fetch results

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\bigquery\client.py:3492, in Client.query(self, query, job_config, job_id, job_id_prefix, location, project, retry, timeout, job_retry, api_method)
   return _job_helpers.query_jobs_query(
       self,
       query,
   (...)
       job_retry,
   )
elif api_method == enums.QueryApiMethod.INSERT:
-> 3492     return _job_helpers.query_jobs_insert(
       self,
       query,
       job_config,
       job_id,
       job_id_prefix,
       location,
       project,
       retry,
       timeout,
       job_retry,
   )
else:
   raise ValueError(f"Got unexpected value for api_method: {repr(api_method)}")

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\bigquery\_job_helpers.py:159, in query_jobs_insert(client, query, job_config, job_id, job_id_prefix, location, project, retry, timeout, job_retry)
   else:
       return query_job
--> 159 future = do_query()
# The future might be in a failed state now, but if it's
# unrecoverable, we'll find out when we ask for it's result, at which
# point, we may retry.
if not job_id_given:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\bigquery\_job_helpers.py:136, in query_jobs_insert.<locals>.do_query()
query_job = job.QueryJob(job_ref, query, client=client, job_config=job_config)
try:
--> 136     query_job._begin(retry=retry, timeout=timeout)
except core_exceptions.Conflict as create_exc:
   # The thought is if someone is providing their own job IDs and they get
   # their job ID generation wrong, this could end up returning results for
   # the wrong query. We thus only try to recover if job ID was not given.
   if job_id_given:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\bigquery\job\query.py:1383, in QueryJob._begin(self, client, retry, timeout)
"""API call:  begin the job via a POST request

See
   (...)
   ValueError: If the job has already begun.
"""
try:
-> 1383     super(QueryJob, self)._begin(client=client, retry=retry, timeout=timeout)
except exceptions.GoogleAPICallError as exc:
   exc.message = _EXCEPTION_FOOTER_TEMPLATE.format(
       message=exc.message, location=self.location, job_id=self.job_id
   )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\bigquery\job\base.py:746, in _AsyncJob._begin(self, client, retry, timeout)
# jobs.insert is idempotent because we ensure that every new
# job has an ID.
span_attributes = {"path": path}
--> 746 api_response = client._call_api(
   retry,
   span_name="BigQuery.job.begin",
   span_attributes=span_attributes,
   job_ref=self,
   method="POST",
   path=path,
   data=self.to_api_repr(),
   timeout=timeout,
)
self._set_properties(api_response)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\bigquery\client.py:833, in Client._call_api(self, retry, span_name, span_attributes, job_ref, headers, **kwargs)
if span_name is not None:
   with create_span(
       name=span_name, attributes=span_attributes, client=self, job_ref=job_ref
   ):
--> 833         return call()
return call()

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\api_core\retry\retry_unary.py:293, in Retry.__call__.<locals>.retry_wrapped_func(*args, **kwargs)
target = functools.partial(func, *args, **kwargs)
sleep_generator = exponential_sleep_generator(
   self._initial, self._maximum, multiplier=self._multiplier
)
--> 293 return retry_target(
   target,
   self._predicate,
   sleep_generator,
   timeout=self._timeout,
   on_error=on_error,
)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\api_core\retry\retry_unary.py:144, in retry_target(target, predicate, sleep_generator, timeout, on_error, exception_factory, **kwargs)
for sleep in sleep_generator:
   try:
--> 144         result = target()
       if inspect.isawaitable(result):
           warnings.warn(_ASYNC_RETRY_WARNING)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\_http\__init__.py:482, in JSONConnection.api_request(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object, timeout, extra_api_info)
   data = json.dumps(data)
   content_type = "application/json"
--> 482 response = self._make_request(
   method=method,
   url=url,
   data=data,
   content_type=content_type,
   headers=headers,
   target_object=_target_object,
   timeout=timeout,
   extra_api_info=extra_api_info,
)
if not 200 <= response.status_code < 300:
   raise exceptions.from_http_response(response)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\_http\__init__.py:341, in JSONConnection._make_request(self, method, url, data, content_type, headers, target_object, timeout, extra_api_info)
   headers[CLIENT_INFO_HEADER] = self.user_agent
headers["User-Agent"] = self.user_agent
--> 341 return self._do_request(
   method, url, headers, data, target_object, timeout=timeout
)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\cloud\_http\__init__.py:379, in JSONConnection._do_request(self, method, url, headers, data, target_object, timeout)
def _do_request(
   self, method, url, headers, data, target_object, timeout=_DEFAULT_TIMEOUT
):  # pylint: disable=unused-argument
   """Low-level helper:  perform the actual API request over HTTP.

   Allows batch context managers to override and defer a request.
   (...)
   :returns: The HTTP response.
   """
--> 379     return self.http.request(
       url=url, method=method, headers=headers, data=data, timeout=timeout
   )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\google\auth\transport\requests.py:538, in AuthorizedSession.request(self, method, url, data, headers, max_allowed_time, timeout, **kwargs)
remaining_time = guard.remaining_timeout
with TimeoutGuard(remaining_time) as guard:
--> 538     response = super(AuthorizedSession, self).request(
       method,
       url,
       data=data,
       headers=request_headers,
       timeout=timeout,
       **kwargs
   )
remaining_time = guard.remaining_timeout
# If the response indicated that the credentials needed to be
# refreshed, then refresh the credentials and re-attempt the
# request.
# A stored token may expire between the time it is retrieved and
# the time the request is made, so we may need to try twice.

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
send_kwargs = {
   "timeout": timeout,
   "allow_redirects": allow_redirects,
}
send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
return resp

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\sessions.py:703, in Session.send(self, request, **kwargs)
start = preferred_clock()
# Send the request
--> 703 r = adapter.send(request, **kwargs)
# Total elapsed time of the request (approximately)
elapsed = preferred_clock() - start

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\adapters.py:486, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
   timeout = TimeoutSauce(connect=timeout, read=timeout)
try:
--> 486     resp = conn.urlopen(
       method=request.method,
       url=url,
       body=request.body,
       headers=request.headers,
       redirect=False,
       assert_same_host=False,
       preload_content=False,
       decode_content=False,
       retries=self.max_retries,
       timeout=timeout,
       chunked=chunked,
   )
except (ProtocolError, OSError) as err:
   raise ConnectionError(err, request=request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connectionpool.py:790, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
response_conn = conn if not release_conn else None
# Make the request on the HTTPConnection object
--> 790 response = self._make_request(
   conn,
   method,
   url,
   timeout=timeout_obj,
   body=body,
   headers=headers,
   chunked=chunked,
   retries=retries,
   response_conn=response_conn,
   preload_content=preload_content,
   decode_content=decode_content,
   **response_kw,
)
# Everything went great!
clean_exit = True

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connectionpool.py:536, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
# Receive the response from the server
try:
--> 536     response = conn.getresponse()
except (BaseSSLError, OSError) as e:
   self._raise_timeout(err=e, url=url, timeout_value=read_timeout)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connection.py:461, in HTTPConnection.getresponse(self)
from .response import HTTPResponse
# Get the response from http.client.HTTPConnection
--> 461 httplib_response = super().getresponse()
try:
   assert_header_parsing(httplib_response.msg)

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py:1395, in HTTPConnection.getresponse(self)
try:
   try:
-> 1395         response.begin()
   except ConnectionError:
       self.close()

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py:325, in HTTPResponse.begin(self)
# read until we get a non-100 response
while True:
--> 325     version, status, reason = self._read_status()
   if status != CONTINUE:
       break

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py:286, in HTTPResponse._read_status(self)
def _read_status(self):
--> 286     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
   if len(line) > _MAXLINE:
       raise LineTooLong("status line")

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\socket.py:706, in SocketIO.readinto(self, b)
while True:
   try:
--> 706         return self._sock.recv_into(b)
   except timeout:
       self._timeout_occurred = True

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\ssl.py:1314, in SSLSocket.recv_into(self, buffer, nbytes, flags)
   if flags != 0:
       raise ValueError(
         "non-zero flags not allowed in calls to recv_into() on %s" %
         self.__class__)
-> 1314     return self.read(nbytes, buffer)
else:
   return super().recv_into(buffer, nbytes, flags)

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\ssl.py:1166, in SSLSocket.read(self, len, buffer)
try:
   if buffer is not None:
-> 1166         return self._sslobj.read(len, buffer)
   else:
       return self._sslobj.read(len)

KeyboardInterrupt: 

fixed_time_boxplot#

A fixted-time boxplot uses fixed-time statistics that rank each point at each time step, and use those to construct confidence intervals for each time step. Always plots the median and interquartile range.